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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Application of: 
Jia Hong Yin 

U.S. Serial No.: To be assigned 
International Application No.: PCT/GBOO/00454 
International Filing Date: 11 February 2000 
International Priority Date: 11 February 1999 

Title: APPAIRATUS FOR, AND METHOD OF, ENCODING INFORMATION 
INTO, AND DECODING INFORMATION FROM, A SEQUENCE OF 
MOVING [MAGES 

PRELIMINARY AMENDMENT 

Box PCT 

Director of the US Patent & Trademark Office 
Washington, DC 20231 

Dear Sir: 

Prior to action in, and calculation of fees for, the above-captioned 
application, please amend the application as follows: 

IN THE SPECIFICATION: 

Kindly amend the specification as follows: 

Clean Specification 

Page 1: 

APPARATUS FOR, AND METHOD OF, ENCODING INFORMATION INTO, 
AND DECODING INFORMATION FROM, A SEQUENCE OF MOVING IMAGES 

BACKGROUND OF THE INVENTION 
1. Field of the Invention 

The present invention relates to an apparatus for, and method of, 
encoding information into and decoding information from a sequence of moving 
images. The technique of coding information into a sequence of video or 
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cinematographic images is also l<nown as electronic watermarking. The 
invention is therefore particularly, but not exclusively, suitable for electronic 
watermarking all types of media having moving images stored thereon or 
involving the transmission or broadcast of moving image data. 

2. Background 

In video broadcasting, data can be in either digital or analogue 
form. Increasingly data is transmitted in digital form. One reason for this is 
because larger amounts of data can be transmitted in digital form than in 
analogue form. Another is because digital signals are less prone to interference 
and can easiily be recovered by using error correction techniques if received 
images are distorted. However, as side effects of facile access to digital data, 
complaints of copyright infringement and for tampering or modification of content 
are increasing. 

Page 2: 

SUMMARY OF THE INVENTION 

According to the present invention there is provided a method of 
encoding information into a sequence of images; the sequence of images 
comprising a first image and a subsequent image; the method comprising the 
steps of: locating an edge in the first image, locating a corresponding edge in a 
subsequent image, comparing relative positions of the corresponding edges in 
said images, thereby identifying at least one relatively moving edge; inserting a 
first code into said moving edge and inserting a second code into said static 
edge. 

Page 4: 

BRIEF DESCRIPTION OF THE DRAWINGS 

Embodiment of the invention will not be described, by way of 
examples only, with reference to the Figures in which: - 
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Page 5: 

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS 

Referring to the Figures, Figures 1 and 2 show block diagrams of 
an encoder 10 and decoder 20 respectively. The invention will now be described 
in the encoding sequence with reference to Figure 1 . The signal is split into two 
portions: a moving image signal 22 and a static image signal 23. This can be 
according to a standard MPEG protocol or a different proprietary image analysis 
(compression) protocol. 
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Version with Markings to Show Changes Made 

Page 1, below the title, please insert heading - BACKGROUND OF 
THE INVENTION --; 

Page 1, above line 6, please insert heading - 1. Field of the 

Invention -; 

Page 1, above line 13, please insert heading - 2. Background - ; 

Page 2, above line 27, please insert heading -- SUMMARY OF 
THE INVENTION -; 

Page 4, above line 27, please insert heading - BRIEF 
DESCRIPTION OF THE DRAWINGS -; 

Page 5, above line 17, please insert the heading - DETAILED 
DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS -. 

IN THE CLAIMS: 

Kindly amend the claims as follows: 

Clean Set of Claims 

3. Apparatus according to claim 1 wherein means is provided to 
identify moving and static edges and signals representative of said moving and 
static image regions are sent via first and second channels so that code is 
inserted into, or recovered from, either or both of said channels. 

4. Apparatus according to claim 1 wherein a first code is inserted 
into signals in a first channel and a second code is inserted into signals in said 
second channel. 

7. Apparatus according to claim 1 wherein code is inserted into a 
moving image region of a moving image sequence in such a way that the code is 
resistant to image compression. 
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9. Apparatus according to claim 1 wlierein the code is inserted into 
boundaries between regions of different luminance, chrominance or contrast. 

11. Apparatus according to claim 1 which is included in video 
recording, video broadcasting, video viewing equipment; or a television receiving 
or a set-top box. 

12. Apparatus according to claim 1, including a digital imager, 
arranged to obtain images of moving objects on a static background and to 
transmit signals indicative of the objects to a location remote from said imager. 
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Version with Markings to Show Changes Made 

Claim 3, line 1 , delete "or 2"; 

Claim 4, line 1 , delete ", or claim 3 when dependent on claim 1 ,"; 
Claim 7, line 1 , delete "any"; 

Claim 9, line 1 , change "claims 1 , 7 or 8" to -- claim 1 --; 
Delete Claim 10; 

Claim 1 1 , line 1 , change "any of claims 1 to 9" to -- claim 1 --; 
Claim 1 2, line 1 , change "any of claims 1 to 9" to - claim 1 -; 
Delete Claim 14; 
Delete Claim 16. 
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REMARKS 



By the amendment, the multiple dependencies have been removed from 
dependent claims 3, 4, 9, 11, and 12 so that each claim depends from a single 
claim and to avoid the fee charged for such claims as originally presented. 



Manelli Denison & Seller PLLC 

2000 M Street, NW 
Suite 700 

Washington, DC 20036-3307 
TEL. (202)261-1020 
FAX. (202) 887-0336 



Dated: Augusts, 2001 



Respectfully submitted, 




William H. Bollman 
Reg. No. 36,457 
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APPARATUS AND METHOD FOR VIDEOSIGNAL WATERMARKING 



The present invention relates to an apparatus for, and metiiod of, encoding information 
into and decoding information from a sequence of moving images. The technique of 
coding information into a sequence of video or cinematographic images is also known as 
electronic watemiarking. The invention is therefore particularly, but not exclusively, 
10 suitable for electronic watermarking all types of media having moving images stored 
thereon or involving the transmission or broadcast of moving image data. / 

In video broadcasting, data can be in either digital or analogue form. Increasingly data is 
transmitted in digital form. One reason for this is because larger amounts of data can be 
15 transmitted in digital form than in analogue form. Another is because digital signals are 
less prone to interference and can easily be recovered by using error correction techniques 
if received images are distorted. However, as side effects of facile access to digital data, 
complaints of copyright infringement and for tampering or modification of content are 
increasing. 

20 

It is known to insert codes into images for the purpose of identifying the owner of the 
images. The images may be in the form of video transmissions, or video clips or stills, for 
example for transmission across a telecommunication link. There are a number of known 
schemes for inserting identification codes into so called 'sync' periods in transmitted 
25 video images. More recently it has been proposed to insert identification codes into the 
image itself in such a way that the code cannot be detected by a human eye. However, 
such schemes may suffer from the disadvantage that low pass filtering and other image 
processes, such as data compression, may remove the code or degrade it to an extent that it 
cannot be recognised. 

30 

UK Patent Application GB-A-2 305 803 (Philips) describes a method of correcting errors 
which occur in a telecine scanning process. The method involves sampling an area of an 
image which includes a structure and comparing this sampled structure with subsequently 
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sampled structures obtained from subsequent image frames. Use of motion vectors 
obtained is then made to correct picture steadiness. 

All current schemes suffer from the disadvantage that coded digital information cannot 
5 easily be transformed and maintain its integrity. Low pass filtering and other processes, 
such as data compression, may occur as a result of image compression algorithms or 
transmission of audio signals across a telecommunication link. Such transforms may 
remove the code or degrade it to an extent where it cannot be recognised. 

10 In the Applicant's published International Patent Application WO-Al-9625005 (Todd), 
there is described a method of coding data into an image. The method of coding and 
decoding information into an image, comprises: dividing the image into MxN blocks, 
selectively encoding (decoding) information into selected blocks in such a way as not to 
be visible to a human eye, wherein in a decoding stage, the decoding is synchronised to 

15 the occurrence of the blocks for analysis of image information. In a preferred embodiment 
the size of insertions and their positions are fixed by processing the image in a block-by- 
block manner, typically with a block size of 8 by 8 pixels. 

In MPEG compression standards, temporal redundancy in image sequences is reduced by 
20 block-based motion compensation. Any change in an image from its previous frame 
caused by object movement is reserved to recover the image in the process of image 
decompression. Whilst previous encoding techniques have been successful, sometimes 
embedded codes have not survived MPEG compression. 

25 The present invention arose to overcome this and associated problems. 

According to the present invention there is provided a method of encoding information 
into a sequence of images; the sequence of images comprising a first image and a 
subsequent image; the method comprising the steps of: locating an edge in the first image, 
30 locating a corresponding edge in a subsequent image, comparing relative positions of the 
corresponding edges in said images, thereby identifying at least one relatively moving 
edge; inserting a first code into said moving edge and inserting a second code into said 
static edge. 
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A corresponding apparatus is also provided. 

The invention thus solves the problem of preserving both static and moving encoded 
5 information duiring subsequent image compression. Another advantage of the present 
invention over the arrangement described in UK Patent Application GB-A-2305 803 
(Philips) is the fact that the present invention is capable of diagrammatically encoding 
watermark code into a signal to be stored, or broadcast. 

10 Preferably the apparatus and method identify moving, and therefore static edges, in an 
image sequence and treat moving and static edges as separate channels each carrying its 
own code(s). Hence identification information is encoded into both moving edges (which 
are not compressed to any extent) and static edges independently. Both channels may 
carry identical codes if required. Alternatively different codes may be inserted into 

15 moving or static edges. The code insertion technique of the present invention may be used 
to embed codes in moving edge pixels. 

Means is advantageously provided to preselect said relative moving edges so that a code 
can be inserted, into switchable sequences. If there is not movement in a sequence of 
20 images, no code is inserted and a different sequence may be selected. 

Preferably coded information is inserted into an image in so called strongly featured 
regions of an image in such a way that the code is resistant to image compression and/or 
low pass filtering, but is not visible to the human eye. Examples of strongly featured 
25 regions of images are textured regions or lines, or edges between two regions of different 
luminance or contrast. In such regions, it is possible to insert a relatively large amount of 
information without significantly altering the image. 

Information encoded into an image may be used for a variety of purposes, for example: 

30 

i) to insert copyright or identification information into video clips or still images; 

ii) to monitor when advertisements or films are played in broadcasts, for monitoring 
royalty payment purposes; 
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iii) to identify a master copy of a data storage medium, sucli as a CD or DVD or video 
disc or similar medium, from which pirated copies may be produced. 

Codes are preferably inserted in edges within an image. Edge regions are known to have 
5 masking properties because of the way the human visual system works. In particular, the 
local orientation of edges are important, and there are specific stmctures in the primary 
visual cortex of the human brain, for detecting the presence of an edge and its local 
orientation. 

10 Coded information is preferably inserted into an image so that it does not alter the local 
orientation of certain features. The insertions are preferably made along the length of a 
local section of edge. The insertions are preferably made as a 2D function, by using for 
example an ellipse which is aligned to the local orientation of the edge. 

15 According to another aspect of the present invention there is provided apparatus for 
decoding infortnation from an encoded moving image sequence, comprising: means for 
identifying a region in an image in the sequence, means for determining whether the said 
region is a moving region or static region and means for recovering code from at least said 
moving image region. 

20 

Corresponding to this further aspect of the invention, there is also provided a method of 
decoding information from an encoded moving image sequence. 
There is correspondingly also provided a method of encoding information into, and 
decoding infonnation from, a moving image sequence, having one or more of the 
25 aforementioned preferred features. 

Embodiments of the invention will now be described, by way of examples only, with 
reference to the Figures, in which :- 

30 Figure 1 shows diagrammatically an embodiment of an encoder according to the 
invention; 

Figure 2 shows an embodiment of a decoder according to the invention; 
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Figure 3 shows a sketch with edges of images in bold; 

Figure 4 shows the sketch of Figure 3, with only moving edges highlighted in bold; 

5 

Figure 5 shows diagrammatically an embodiment of the invention, incorporated in a 
surveillance camera; 

Figure 6 is a diagrammatical view of a supermarket in which the surveillance camera of 
10 Figure 5 may be used; 

Figure 7 is an example of an image of pedestrian flow through a supermarket check-out 
area; and 

15 Figure 8 is a vector diagram representative of the pedestrian flow of Figure 7, 

Referring to the Figures, Figures 1 and 2 show block diagrams of an encoder 10 and 
decoder 20 respectively. The invention will now be described in the encoding sequence 
with reference to Figure 1. The signal is split into two portions: a moving image signal 22 
20 and a static image signal 23. This can be according to a standard MPEG protocol or a 
different proprietary image analysis (compression) protocol. 

A first code is inserted into the static image signal and a second code is inserted into the 
moving image signal. These signals are then transmitted via separate channels. 

25 

The moving edge encoding sequence is now described with reference to the following 
Equations 1 to 3 and Figures 3 and 4. 

Moving edges and static edges may be defined as the following: 

30 

Let li(x,y), hix.y) and h(x,y) denote three consecutive images in an image sequence. Let 
E2(x,y) be an edge image from image hix.y). The moving edge image ME(x,y) is defined 
as follows; 
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ME(x,y) = Du(x,y).E2(x,y).D23(x,y) - (Eqn 1) 
where Di2ix,y) = \li(x,y)-l2(x,y)\, D2s{x,y} = \h(x,y) - h(x,y}\. - (Eqn 2) 

5 The static edge image SE(x,y) may be obtained using the following equation: 

SE(x,y) = ; E2(x,y), ifME(x,y) = 0 

Once inserted the coded information is treated the same as a moving image by any 
10 compression algorithm or protocol such as MPEG. Static code is included in early 
sequences of a series of encoded images so that when compressed the encoded 
information remains in tact. 

Brief reference will now be made to Figure 2, which depicts a decoder 20. Decoder 20 
15 receives encoded images signal 24. The image signal 24 is split into a moving image 
signal 26. Edge diffraction then occurs. Edge detector 27 detects edges in moving 
images. Edge detector 28 detects edges in static images. Detected edge signals 29 and 30 
are subtracted one from another at subtracter 3 1 and a static edge is decoded at static edge 
decoder 32. Moving edges are decoded directly from a signal presented by moving edge 
20 detector 27 to the moving edge decoder 33. Static edge decoded signals 34 and moving 
edge decoded signals 35 are added at 36 to provide an electronic watermark signal which 
indicates for example, the origin of a signal or the owner of copyright in an image 
sequence or piece of video footage. 

25 The invention may be used to encode information onto all forms of recording media. 

These may include videotape, video disc, compact disc (CD or DVD), or any other form 
of video storage medium. Similarly the invention may be incorporated into video 
broadcasting systems, video editing equipment, video monitoring equipment, televisions, 
computers or any other piece of electronic equipment used to produce or view video 

30 images, including a video cassette recorder and/or a set-top box. 

The invention has been described, by way of example only, and it will be understood that 
variation may be made to the embodiments described without departing from the scope of 
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the invention. In an alternative embodiment, shown in Figures 5 to 8, the invention may 
be deployed within shops or other public places in order to monitor activity of purchasers 
or members of the public. The invention allows for the automation of data capture relating 
to motion of, say, shoppers and optionally, its subsequent encoding for transmission. 

5 

Figure 6 represents the layout of a typical supermarket. The system to be implemented 
uses real-time video data from appropriately positioned Close Circuit Television (CCTV) 
cameras to monitor the customers arriving at entrance 102, check-outs 106 and exit 104. 
The system uses information extracted from video data to estimate net arrival/departure 
10 rate of customers and from this maintains an estimate of the number of customers inside 
the store at any time. This together with the information from the aforementioned areas 
enables the system to maintain estimates of the number of customers: 

a) within the store shelf area 
15 b) queuing at the check-outs 

In addition a correlation between the number of customers arriving at the check-outs and 
the number of customers arriving at the store at some earlier time allows the system to 
adaptively optimise parameters of a prediction model of the store. This optimised 
20 prediction model enables the system to estimate impending customer demand for check- 
outs in advance of the need becoming apparent. This allows for timely staff re- 
deployment to avoid any reduction in customer service. 

Output from the system is preferably in two main forms: 

25 

1. Real-time data concerning conditions in the store as they happen. This data in turn 
may be presented either 

a) On a screen in some management area, or 

b) As immediately useful instructions relayed through radio headsets. These 
30 instructions need to be unambiguous. It is envisaged at this stage that the 

instructions consist of messages to open or close a number of check-outs. 

2. The system will be capable of producing a statistical report concerning the arrival 
rate of customers and the length of time they stay in the store. This data will be 
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suitable for planning overall staff levels and will be a useful source of marketing 
information. 

Imaging equipment may be used in conjunction with a niiicroprocessor dedicated to 
5 determining length of queues or regions of a store visited by an unusually large or small 
number of customers. This may be particularly advantageous in predicting employee 
allocation at check-outs for example. Similarly the system may be configured to monitor 
automatically a specific region of a store or even a specific aisle or display. By adapting 
control and analytical software the system may be arranged to provide a statistical 
10 assessment of how popular or unpopular a particular type of marketing or location of a 
product is performing. 



30 
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CLAIMS 



5 1 . Apparatus for encoding information into a moving image sequence, comprising: 

means for identifying a region in an image in the sequence, means for determining 
whether the said region is a moving or static region and means for inserting a 
coded into at least said moving image region. 

10 2. Apparatus for decoding information from an encoded moving image sequence, 

comprising: means for identifying a region in an image in the sequence, means for 
determining whether the said region is a moving or static region and means for 
recovering code from at least said moving image region. 



25 



3. Apparatus according to claim 1 or 2 wherein means is provided to identify moving 
and static edges and signals representative of said moving and static image regions 
are sent via first and second channels so that code is inserted into, or recovered 
from, either or both of said channels. 

4. Apparatus according to claim 1, or claim 3 when dependant on claim 1, wherein a 
first code is inserted into signals in a first channel and a second code is inserted 
into signals in said second channel. 

5. Apparatus according to claim 4 wherein the first and second codes are identical. 

6. Apparatus according to claim 4 wherein the first and second codes are different. 



7. Apparatus according to any claim 1 wherein code is inserted into a moving image 
region of a moving image sequence in such a way that the code is resistant to 
30 image compression. 



8. Apparatus according to claim 1 wherein code is inserted into a region of a moving 
image so that the code is resistant to low pass filtering. 
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9. Apparatus according to claims 1, 7 or 8 wherein the code is inserted into 

boundariies between regions of different luminance, chrominance or contrast. 

5 10. Apparatus substantially as herein described and with reference to the Figures. 

11. Apparatus according to any of claims 1 to 9 which is included in video recording, 
video broadcasting, video viewing equipment; or a television receiving or a set-top 
box. 

10 

12. Apparatus according to any of claims 1 to 9, including a digital imager, arranged to 
obtain images of moving objects on a static background and to transmit signals 
indicative of the objects to a location remote from said imager. 



13. Apparatus according to claim 12 operating under control of software for 
statistically analysing said code, so as to provide an indicator of the amount of 
movement of the objects during a chosen time interval. 

14. A method of encoding information into, or decoding information from, a moving 
image sequence using the apparatus of claims 1 to 9. 

15. A method of encoding information into a sequence of images; the sequence of 
images comprising: a first image and a subsequent image, the method comprising 
the steps of: locating an edge in the first image, locating a corresponding edge in 
the subsequent image, comparing relative positions of the corresponding edges in 
said first and second images, thereby identifying either a relatively static edge or 
two relatively moving images; inserting a first code into said moving edge and 
inserting a second code into said static edge. 

16. A medium storing a video or cinematographic image or sequence of images 
thereon, characterised in that a code has been inserted into selective portions of 
said images, according to the method of claim 14, when dependant on claims 1 or 
claims 4 to 9, or claim 15. 
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IN THE UNITED STATES 
PATENT AND TRADEMARK OFFICE 



Declaration and Power of Attorney (Sole Inventor) 



As a below njuned inventor, I hereby declare that; 

My residence, post office address and citizenship are as stated below next to my name. 

I believe that I am the original, first and sole inventor of the subject matter which is claimed 
and for which £i patent is sought on the invention entitled Apparatus for, and metho d 
of, encoding liaf ormatlon Into, and decoding xnf ormatiioTiL from, a sequence of 
moving images 

the specification of which; Q is being filed herewith. 



I hereby state that I have reviewed and understand the contents of the above identified 
specification, including the claims, as amended by an amendment, if any, specifically referred to 
in this oath or declaration. 

I acknowledge the duty to disclose all information known to me which is material to 
patentability as defined in Title 37, Code of Federal Regulations, 1.56. 

I hereby claim foreign priority benefits under Title 35, United States Code, 119 of any 
foreign application(s) for patent or inventor's certificate listed below and have also identified 
below any foreign application for patent or inventor's certificate having a filing date before that 
of the application on which priority is claimed: 



I hereby claim the benefit under Title 35, United States Code, 120 of any United States 
appIicadon<s) listed below and, insofar as the subject matter of each of the claims of this 
application is not disclosed in the prior United States application in the manner provided by the 
first paragraph of Title 35, United States Code, 112, I acknowledge the duty to disclose all 
information known to me to be material to patentability as defined in Title 37, Code of Federal 
Regulations, 1.56 which became available between the filing date of the prior application and the 
national or PCX international filing date of this application: 
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I hereby declai-e that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were rnade with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States 
Code and that such willful false statements may jeopardize tlie vaUdity of the application or any 
patent issued thereon. 

I hereby appoint the following attorneys with full power of substitution and revocation, to 
prosecute said appUcation, to make alterations and amendments therein, to receive the patent, and 
to transact all business in the Patent and Trademark Office connected therewith: 

William H. Bollman (Reg. No. 36AW 

W. Wairen Taltavull (Reg. No. 



b ] Edward J. Siemberger (Reg. No. .36,017) 

Jeffrey S. Melcher (Reg. No. 35.950) 

Leon R. Turkevich (Reg. No. 34035) 

Paul E. White, Jr. (Reg. No. 32,011) 

Please address all correspondence to: 

WiUiamH . Bollman 
MA^^LLIJ PEMSQN ASELTER PLLC 
2000Mj teeet,N.W. Suite7QO 
Waslimg ton^DC_ 2Q036-33 07_ 
TEL: (202)261-1000 
FAX: (202)887-0336 

Please direct all telephone calls to: 

William H. Bollman at (202) 261-1020. 

Full name of sole inventor: Jia Hong Yin 

Tny^nmr-.^iotn^hii^. ^^^^^- ^^C^^ Date: 1^1^ 

Residence: FulhamT London, SW^ZRW, England E^Q 
Citizenship: British 

Post Office Addlress: 3 Watenneadow Lane, Fulham, London, SW6 2RW, England 
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